EMT Practice Test

1. Question Content...

---

Question1: An ecommerce brand runs a multi-cell Conversion Lift test. The brand needs to determine if bidding in the Facebook auction based on user value calculated from its LTV model versus demographic targeting improves performance by 10%. The p-value for the test is calculated as p = 0.95.
How should the analyst interpret bidding based on user value?

Question2: An advertiser recently ran a month-long campaign on a new media platform. This campaign targeted customers who had purchased from the advertiser in the past year. Of the 10 million customers targeted, 3 million were reached. The average frequency for the campaign was three impressions over the month. The advertiser spent $100,000 on this media buy.
After the campaign, an analyst from the media platform noticed that customers who received six or more impressions were twice as likely to purchase than those who received three or fewer impressions. To increase the number of users who receive six or more impressions, the analyst recommends that the advertiser double their spend. The goal is to increase the frequency from three to six in order to drive a significant increase in incremental return on ad spend.
What primary concern should the advertiser's in-house measurement team have about this conclusion?

Question3: An analyst analyzes 1,000 Facebook video campaigns and calculates there is a correlation of r = 0.6 between view duration of a campaign and additional ad recall lift. What should the analyst tell the company about the relationship between view duration and ad recall lift?

Question4: A new ecommerce company that sells raincoats for dogs leverages Facebook to drive product sales. The company serves a variety of videos and static images and only targets people who previously visited their website. The company historically spends $50,000 per quarter and always hits its ROAS goal of 3x.
Last quarter, the company aggressively tried to grow its business and tripled spend. At the end of last quarter, they compared performance between the previous two quarters and noticed the following differences in performance metrics:
* ROAS decreased by 40%
* Average frequency increased by 50%
* CPMs increased by 60%
* Reach increased by 10%
* Cost per click decreased by 15%
* Allocation of budget between creative formats remained the same
An analyst needs to run a test to identify how to scale spend efficiently. Which variable should the analyst test first?

Question5: A start-up ecommerce brand that sells pet products wants to test campaign structure. It would like to determine if it should have separate ad sets targeting different pet interest groups or consolidate all interest groups into one ad set.
The brand sets up a multi-cell Conversion Lift test for one month. At the end of the test, no results are available to review, due to insufficient statistical power.
Which two approaches should the analyst recommend? (Choose 2)

Question6: A brand needs to measure how frequency affects conversions and ROI. Currently the brand runs a campaign on 2x frequency for seven days. How should the brand set up a test to understand how frequency affects incremental conversions?

Question7: An ecommerce brand is interested in understanding which approach delivers the highest ROAS. The advertiser is leveraging two distinct ad sets.
Ad Set A:
Budget: $100,000 Audience: Men, ages 45+ Creative: Product Focused
Ad Set B:
Budget: $120,000 Audience: Men, ages 18-44 Creative: Product Focused
Which two approaches could be recommended to analyze these campaigns? (Choose 2)

Question8: A large gaming company is developing a measurement strategy designed to shift its planning and buying on Facebook. The company believes that its ability to iterate on testing will provide more long-term growth than the short-term sales boosts from individual campaigns. The conversion rate is high enough that they should have NO issues with statistical power at any size of test/holdout groups. It is planning on running multi-cell Conversion Lift tests on a monthly basis.
What test/holdout proportions should the company use for each test cell?

Question9: A taxi company is working on building an understanding of household customer lifetime value. Some of their customers order via digital platforms, some via phone, and some alternate between the two. They currently calculate lifetime value (LTV) by looking at all hashed order data including email addresses for online customers and all phone numbers for phone orders. The results showed that email customers had a yearly LTV of $100, and the phone customers had a yearly LTV of $80. However, the company is aware that a group of people are introducing some noise into the results by ordering via both phone and online.
What solution should the analyst recommend to enhance the ability to create a functional LTV model?

Question10: An advertiser obtains a p-value of 0.20 after a matched pairs multi-cell experiment. The objective was to measure whether advertising strategy A generated more incremental business outcomes than advertising strategy B.
The test had high statistical power probability, and the result was not a false negative.
What could the advertiser conclude?

Question11: An advertiser runs a campaign that yields the following results:

The results are statistically significant. What is the incremental ROAS for each test?

Question12: An analyst reviews Conversion Lift test results mid-flight and has the option to take action immediately. The conversion cycle for this advertiser is 14 days, and the advertiser is running a multi-cell Conversion Lift test with equal budgets between both cells:
* Strategy A: Auction buying; Automatic Placements
* Strategy B: Auction buying; Facebook News Feed only
After the first day, the results are as follows:
* Strategy A: Automatic Placements: 12 conversions
* Strategy B: Facebook News Feed only: 14 conversions
What should the analyst recommend?

Question13: A car manufacturer discovers that the purchase journey is typically one-year long, involves several media channels and is followed by a dealership visit that ends with a purchase at the dealership. Historically, sales are mostly influenced by the quality of the customer service experience. The manufacturer has a KPI of driving incremental customers to its website.
What measurement solution should be used?

Question14: A longitudinal data set is missing values.
Which approach should be used to minimize bias in a forecast considering it is a small sample?

Question15: An analyst wants to measure a campaign that is delivering across Facebook and Paid Search. The client wants to understand the causal sales impact of each channel. Which methodology should the analyst use?

Question16: A financial services brand is launching a new tagline. The primary goal is to drive cost-efficient leads. The brand also wants to prove that its audience is able to associate the new tagline back to the brand. The brand uses both the Brand Awareness and Lead Generation campaign objectives.
A multi-cell test was launched to measure how effectively its campaigns increased message association and drove leads. Eighty percent of the budget that was measured during the time period was allocated toward the Brand Awareness objective. The results estimated a significant lift in message association but flat results on leads.
Which action should the brand take to meet its primary goal?

Question17: An online retailer knows their incremental effect from Facebook ads from their previous Conversion Lift test. They want an always-on attribution solution that allows them to allocate its budget across publishers on an ongoing basis. The challenge is identifying a model that is as close to their true business value as possible.
Refer to the following results:

Which Attribution model should the retailer choose?

Question18: An advertiser wants to conduct a multi-cell Brand Lift test. The cells are designed to help the advertiser understand whether adding Instagram Stories as a placement impacts brand awareness. The test is designed with the following:
* Facebook Feed + Instagram Feed
* Instagram Stories + Facebook Feed + Instagram Feed
* Automatic Placements
What should the analyst recommend based on the test results?

Question19: An analyst has calculated an average lift from a random sample of 300 similarly sized campaigns. The analysis showed the following:
* Minimum lift is 98
* Maximum lift is 177
* 95% confidence interval for the average lift per campaign is (133, 145)
* Average p-value for each individual campaign's lift is p = 0.03
What should the analyst conclude about the distribution of lifts?

Question20: A cosmetics company advertises predominantly on Facebook and Instagram. The company wants to test alternative creative messaging for holiday season campaigns targeted to a narrow audience to discover which message achieves the most efficient sales results.
Which measurement approach should the company use while minimizing CPM impact?

Question21: A local retailer wants to measure the effectiveness of Facebook campaigns based on the number of in-store purchases after customers have been exposed to an ad within the platform.
Where should outcome data be sourced?